Skip to content

Conversation

@jdoucett
Copy link
Contributor

Summary

Implements Phase 3 of the fortress-archive engine — leveled incremental backup (L0 full baseline + L1/L2... delta tars containing only files changed since the last checkpoint) — per docs/RFC_incremental_v2.md §3, previously marked [DEFERRED - DORMANT]. Built via an approved implementation plan, phase by phase (each phase's tests green before the next started), then independently reviewed by two models (Opus + Fable 5) with confirmed findings fixed.

Pilot target: repository_X1D_2_spectral-standoff (the RFC's own reference case - 2.2 TB re-shipped for 2 of 36,186 changed files under the old whole-tar path).

What's in each commit

  • P3-1 (archive.py): the level-decision engine - effective_catalog (folds L0 + ordered deltas), decide_level (baseline/append/modify/delete/NOOP/forced-rebaseline), write_level_catalog, badge_lock (advisory per-badge flock across the full decide+ship+manifest-write critical section), level_and_ship, a kind= extension to manifest_path/load_manifest/write_manifest so a Phase-2 and Phase-3 manifest can coexist for one badge without colliding, an extra_record_fields= extension to send_to_fortress, and the incremental config flag (mutually exclusive with size_routing).
  • P3-2 (reclaim.py): verify_chain (the leveled sibling of verify_target, reconstructing expected state via effective_catalog), is_leveled_manifest, a manifest_for_project fix to deterministically prefer a leveled manifest when both schemas exist for one project, and --scan/single-config wiring so leveled targets never reach the legacy per-log verify().
  • P3-3 (restore.py): select_chain/check_chain_complete/extract_level/apply_deletions/fold_expected/restore_leveled_target, implementing the RFC §3.4 gate order exactly (completeness gate before any transfer, empty-dest guard, overlay + deletion replay in strict order, mandatory final reconciliation). --to-timestamp for point-in-time restore. Existing Phase-2 restore path untouched.
  • Fix commit: addresses every confirmed finding from the two-reviewer adversarial pass (see below).

Review findings fixed

Two independent reviewers (Opus, Fable 5) each reviewed the full diff against the plan and RFC invariants, without seeing each other's findings. Real issues caught and fixed:

  • Data-integrity bug: the manifest recorded the decision member set for a level rather than what actually shipped - a file vanishing between enumerate and zip would have permanently poisoned that chain for restore. Fixed: catalog now built from the zip's real namelist; raises loudly on any discrepancy.
  • Precision regression: restore's CLI resolved a leveled manifest by project-name string match instead of exact badge - could silently pick the wrong target's manifest if two configs shared a project name. Fixed: exact-badge lookup restored.
  • Invariant violation: reclaim.py's single-config mode (not just --scan) never got leveled-target routing, so it would perpetually false-DRIFT a leveled target. Fixed.
  • Operational cost regression: L0 ships lost the existing stale-zip resume guard entirely, meaning a failed baseline upload of a multi-TB target would redo the whole zip on retry. Fixed: resume restored for L0 specifically; deltas still correctly skip it (RFC §3.5.1).
  • Also fixed: verify_chain's completeness gate was scoped to every historical level rather than just the current chain; the NO-OP path was firing the T1 "ACT NOW" Teams bell on routine no-op runs (downgraded to email-only per the project's own notification-triage policy); tar_bytes was never populated (an RFC §3.3 day-one field); duplicate completion emails per leveled ship; a raw FileNotFoundError instead of a named abort on a missing catalog sidecar; missing documentation (module docstring, CLAUDE.md, config.example.json, RFC status flip from DEFERRED to BUILT); and a stale branch base (both reviewers independently caught this - merged in origin/main's f4be033 cleanly).

Explicitly out of scope

P3-4 (retain-N-L0 space reclaim, guarded hsi rm, hsi ls -j hardening) - a documented follow-up, not part of this build.

Test plan

  • Full suite: python3 -m pytest tests/ -q - 392 passed (312 pre-existing baseline + 80 new across tests/test_incremental_engine.py, tests/test_leveled_reclaim.py, tests/test_leveled_restore.py, plus fix-stage additions)
  • Two independent model reviews (Opus, Fable 5), findings fixed, re-verified independently
  • Branch merged with current origin/main (f4be033), verified up to date and green just before this push
  • Not yet done (follow-up, needs a human decision on timing): manual dry-run on Negishi - convert repository_X1D_2_spectral-standoff's config from size_routing: true to incremental: true, run once for L0, touch 1-2 files, run again, confirm a tiny delta ships (not a full re-ship) and reclaim.py --scan reports SAFE via verify_chain
  • Not yet done (separate repo, out of scope here): florasense-tools repo_backup.py's three-way dispatch needs extending to recognize config["incremental"] before this can actually run from the existing deployment wrapper

🤖 Generated with Claude Code

…delta ship)

Implements docs/RFC_incremental_v2.md §3's P3-1 slice in archive.py: L0 full
baseline + (mtime,size)-diffed delta levels, so a run re-ships only what
changed since the last checkpoint instead of a whole target or whole shard.

New in archive.py:
  - effective_catalog(manifest, log_dir, upto_quaver=None) — folds a leveled
    manifest's L0 + deltas into one (mtime,size) catalog; shared by
    decide_level here and by reclaim's verify_chain / restore's fold_expected
    in later slices.
  - decide_level(badge, live_catalog, manifest, force_full, log_dir) — full
    decision matrix: baseline, append, modify-only, delete-only/deletions-
    only, true NO-OP, forced-rebaseline (byte-sum guard), force_full.
  - write_level_catalog(log_dir, run_timestamp, catalog_subset) — atomic
    sibling catalog file + sha256, keeps the manifest lean.
  - record_noop_event / write_noop_log — manifest event + a
    status:"noop","leveled_object":True run-log pair so reclaim's scan can
    skip it (P3-2).
  - badge_lock(vault_dir, badge) — blocking fcntl.flock context manager held
    across the whole decide+ship+manifest-write critical section.
  - level_and_ship(...) — the leveled analog of route_and_ship; NOOP ships
    nothing, deletions-only ships no tar, otherwise ships via an injected
    ship_level(level_obj, stem) and writes the manifest atomically.
  - manifest_path/load_manifest/write_manifest gain kind="routing"|"leveled"
    so a badge converting from Phase-2 size-routing to Phase-3 leveling never
    clobbers the other schema's manifest file.
  - send_to_fortress gains extra_record_fields=None, merged into run_record at
    construction (behavior-neutral default) — Phase 3 tags
    leveled_object/badge/level/blonde/parent_blonde on a level's run-log.
  - load_config: incremental (default False), mutually exclusive with
    size_routing, and the routing_log_dir_ok durability guard now covers
    either flag.

__main__ hook points: a parallel `incremental + --globus` refusal beside
size_routing's; a parallel `if incremental:` ship block (enumerate ->
level_and_ship -> completion/NO-OP email) before the whole-target fallback;
`--full` folded into the existing --fresh/--force-resume CLI handling to
force a fresh L0 baseline. The level-aware resume guard is resolved by
construction: the incremental ship_level closure always re-zips from source
(only_arcnames=members) and never calls find_existing_zip, so a failed delta
simply re-zips fresh on retry instead of misfiring the whole-target
stale-zip guard.

Hard invariant (RFC §3.1): a same-size/same-mtime in-place rewrite is
invisible to the cheap diff. decide_level's NO-OP path also checks the live
byte-sum against the effective catalog's total (forcing a full-content delta
on a genuine mismatch); a true NO-OP emails loudly via the existing
send_alert_email/notify_bell with the RFC's exact required wording.

Tests: new tests/test_incremental_engine.py (38 tests) covering
effective_catalog folding (L0-only, +1 delta, +N overlapping modify/delete,
upto_quaver cutoff), decide_level's full decision matrix, write_level_catalog
round-trip+hash, the manifest kind= collision test (Phase-2 and Phase-3
manifests for the same badge coexisting), badge_lock contention, and
level_and_ship orchestration (baseline/append/delete-only/NOOP/resume-after-
failure) via an injected ship_level recorder. tests/test_incremental_primitives.py
(PR#8 primitives) is untouched. Full suite: 350 passed (312 existing + 38 new).

P3-2 (leveled reclaim / verify_chain) and P3-3 (restore.py leveled support)
are separate follow-up slices per the plan, not part of this change.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Implements docs/RFC_incremental_v2.md section 3.5.2's P3-2 slice in reclaim.py:
the leveled sibling of verify_target(), so a Phase-3 leveled-incremental
target's delta-level run-logs no longer perpetually false-DRIFT once the
first delta ships.

New in reclaim.py:
  - is_leveled_manifest(m): discriminates a Phase-3 manifest (schema
    "leveled_v1" / a "levels" list) from a Phase-2 size-routing manifest
    (flat "objects" list).
  - verify_chain(manifest, holds, log_dir, ...): reconstructs the expected
    state via archive.effective_catalog (L0 union deltas, add/modify applied,
    deletions removed) rather than a per-log member set; requires every
    level's non-null fortress_tar present on tape (any missing one becomes
    FAILED "chain is broken - not restorable", never a partial SAFE);
    per-arcname newer-than-archived is judged directly against the
    reconstructed catalog's own (mtime,size), floor_s on both sides, matching
    decide_level's own comparison. Same verdict cascade and INV-E exclusion
    re-gate as verify()/verify_target().
  - manifest_for_project fix: a badge mid-conversion from Phase-2 to Phase-3
    can have both a routing manifest and a leveled manifest for the same
    project at once (the whole point of manifest_path's kind= split). Glob
    order doesn't reflect which is current, so this now deterministically
    prefers the leveled manifest when both exist.

--scan wiring: the existing routed_object per-log skip now also skips
leveled_object logs (delta/NO-OP level logs, covered by their manifest
instead); find_manifests() results are split into non-leveled
(verify_target, unchanged) and leveled (verify_chain) sets, and the --match
scope filter covers both.

Bug found + fixed along the way: archive.write_level_catalog's sibling
catalog sidecar files live directly in log_dir next to real run-logs (by
design, so effective_catalog can resolve a relative catalog_ref) - but
find_logs' bare *.json glob also matched them (no "project"/"status"
fields), surfacing as a bogus [FAILED] None entry in every --scan report
once any target uses leveled incrementals. Fixed by excluding
*.catalog.json from find_logs' glob.

Tests: new tests/test_leveled_reclaim.py (21 tests) covering
is_leveled_manifest, verify_chain's full verdict cascade (SAFE
baseline-only, SAFE across a real L0+delta chain, DRIFTED via
count/bytes/newer/INV-E, FAILED via a missing intermediate tar and via no
levels, GONE, HOLD, CANONICAL), manifest_for_project's Phase-2/Phase-3
disambiguation, and an end-to-end --scan integration test proving a
leveled_object log is skipped and its target instead gets one aggregate
verdict from verify_chain while an unrelated legacy project's log still
flows through verify() unaffected. tests/test_routing_reclaim.py is
untouched. Full suite: 371 passed (350 existing + 21 new), zero
regressions.

P3-3 (restore.py leveled support) is a separate follow-up slice per the
plan, not part of this change.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Implement full/point-in-time restore for a Phase-3 leveled-incremental
target, per docs/RFC_incremental_v2.md §3.4 and the plan's P3-3 table.

New in restore.py (Phase-2 restore_target/select_objects/CLI path untouched):
  - select_chain(manifest, to_quaver=None): L0 + its deltas (parent_blonde ==
    current_l0_blonde), sorted by quaver, optionally cut off at a point in
    time (mirrors archive.effective_catalog's upto_quaver exactly).
  - check_chain_complete(chain): gate BEFORE any transfer -- every non-null
    fortress_tar in the chain must be on tape, else "chain is broken", never
    a partial restore.
  - extract_level(node, dest, tmp_dir): Path A extraction -- hsi get -> tar xf
    -> zipfile.ZipFile(...).extractall(dest); extractall's overwrite-same-name
    behavior on the accumulating tree IS the overlay.
  - apply_deletions(node, dest): idempotent removal of a level's recorded
    deletions.
  - fold_expected(manifest, log_dir, to_quaver=None): the mandatory final-
    reconciliation target, via archive.effective_catalog's own L0+delta fold
    -- the same fold decide_level diffs against and reclaim.verify_chain
    reconstructs from.
  - verify_level_members(node, dest, log_dir): per-level MD5 verify against
    that level's own run-log (object_runlog_checksums reused verbatim).
  - restore_leveled_target(...): orchestrates the RFC §3.4 gate order exactly
    -- select_chain -> check_chain_complete -> check_dest -> per-node
    extract+verify+apply_deletions (deletions-only nodes skip extraction
    entirely) -> final walk_restored(dest) == fold_expected(...) or ABORT.

CLI: main() now resolves a config to whichever manifest schema the target
was actually archived with (reclaim.manifest_for_project, reusing P3-2's
leveled-preference fix) and dispatches on reclaim.is_leveled_manifest;
added --to-timestamp for point-in-time restore. --slot/--member remain
Phase-2-only (rejected for a leveled manifest with a clear error).

New tests/test_leveled_restore.py (8 tests), reusing test_restore.py's
fake_extract-closure/hsi_exists-monkeypatch conventions and
test_leveled_reclaim.py's manifest-fixture style: full L0+2-delta chain
restore (incl. a real deletion) reconstructs the exact tree; a missing
intermediate tar aborts before any extraction; a deletions-only node
applies its removal with zero extractor calls (asserted explicitly); a
per-member MD5 mismatch aborts; a final-reconciliation mismatch aborts
(a stray file no per-level check catches); --to-timestamp stops the chain
at the right point; a non-empty dest is refused; an empty-levels manifest
aborts cleanly.

Full suite: 379 passed (371 existing + 8 new), zero regressions.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The branch was based on e41a772, one commit behind origin/main. Without
this merge, HSI_PATH_DIRS and the regression test
test_fake_binaries_win_even_when_host_has_real_hsi_bin_dirs were absent
from this branch's tree, and the local suite run was exercising the old
inline ship_object PATH literals instead of the fixed version.
Fixes confirmed bugs and invariant violations from the Opus/fable-5 review of
feat/phase3-leveled-incremental, plus in-scope deviations from the plan:

- reclaim.py: single-config invocation (`python reclaim.py config.json`) had
  no `incremental` branch and fell through to legacy per-log verify(), which
  perpetually false-DRIFTs/FAILs a leveled target. Now routes to verify_chain
  via the exact leveled manifest, mirroring the --scan path.
- archive.py (level_and_ship): the manifest catalog was built from the
  DECISION member set, not what actually shipped. make_zip_files silently
  drops any only_arcnames entry that vanished between enumerate and zip, so a
  race could poison the manifest with a file never on tape, permanently
  breaking restore.verify_level_members for that chain. ship_level now
  reports the zip's real namelist (zip_members); level_and_ship builds the
  catalog from it and raises immediately on any discrepancy instead of
  silently recording a phantom member.
- restore.py: config-mode manifest lookup used manifest_for_project (matches
  by project-name substring across the whole vault), which could silently
  resolve to the WRONG target's manifest when two configs share a
  project_name but differ in source_folder/file_pattern (distinct badges).
  Now resolves by the config's own exact badge, preferring the leveled
  manifest, matching manifest_for_project's tie-break without widening the
  match.
- reclaim.py (verify_chain): the tape-completeness gate checked EVERY tar in
  manifest["levels"], including an older, superseded L0+deltas left behind by
  a --full rebaseline. Scoped to the current chain only (matching
  effective_catalog/select_chain's own node set) so a superseded chain's tar
  going missing (once tape pruning exists) can't false-FAIL a target whose
  restorable chain is intact.
- archive.py: the leveled NO-OP path fired the T1 "ACT NOW" Teams bell on
  every routine unchanged run; the bell is documented (CLAUDE.md) as reserved
  for FAILED/STALE/EXCLUSION-ABORT. Downgraded NO-OP to email-only.
- archive.py: send_to_fortress now returns the retrieved tar's on-tape size
  (tar_bytes) instead of always None on the leveled path (RFC §3.3 reclaim-
  readiness field).
- archive.py: send_to_fortress gained quiet_success= to suppress its own
  completion email for leveled ships, since __main__'s incremental branch
  already sends one summary email per run (was doubling up).
- archive.py: L0 leveled ships now attempt find_existing_zip resume (only
  L0 — deltas still always re-zip fresh by construction), closing an
  operational-cost regression where a failed L0 upload on a multi-TB pilot
  target would re-zip the entire source on every retry. Staleness decision
  factored into a pure, unit-tested l0_zip_is_stale() using the survivor
  catalog already in hand.
- restore.py: verify_level_members now raises a named RestoreError (not a
  raw FileNotFoundError) if a level's sibling catalog_ref file is missing.
- Merged origin/main (f4be033) — the branch was one commit stale, and its
  diff against origin/main was silently reverting the HSI_PATH_DIRS
  PATH-shadowing decoy-binary fix and its regression test.
- Documentation: module docstring (archive.py), CLAUDE.md (config key table +
  new "Leveled incremental (Phase 3)" section), config.example.json, and
  RFC_incremental_v2.md §3 status header now reflect the shipped feature
  instead of reading DEFERRED/undocumented.

Findings reviewed but not changed:
- P3-1 manifest schema doc gaps were closed above rather than left as a
  follow-up.
- Documentation of Phase-3 in CLAUDE.md/RFC previously described the feature
  as deferred; this is now corrected in the same change rather than deferred
  further.

Full suite: 392 passed (was 384 pre-fix; +8 new/expanded test cases).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@jdoucett Jarrod S Doucette (jdoucett) deleted the feat/phase3-leveled-incremental branch July 19, 2026 15:05
Sign in to join this conversation on GitHub.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant